A Grammar for Language and Co-verbal Gesture

Meaning in everyday communication is conveyed by various signals including spoken utterances and spontaneous hand gestures. The literature has attested that gestures function in synchrony with speech to deliver an integrated message, or a “single thought” [5], [1], exhibit language-specific properties [2] and are subject to formal semantic modeling [3]. One of the challenges in modeling synchrony is to use the form of the verbal signal, the form of the gesture and their relative timing to produce an integrated meaning representation. We meet this challenge by exploiting well-established semantic composition rules for deriving meaning from the form of the multimodal action. So, while the existing grammars (HPSG, LFG, CCG) produce semantic representations for unimodal input, we argue that any formalisation of language should fit into the multimodal perspective of synchronising language and co-verbal gesture. The temporal performance of the gesture, its dimension(s) and ambiguous form do not uniquely determine the synchronous linguistic phrase. This raises the distinction between synchrony and simultaneity, thereby allowing for a gesture to attach to more than one constituent without causing ungrammaticality or incoherence. Nevertheless, the choices of attachment are constrained. We consider (1) ill-formed despite the semantic relation between the container shape of the hands and the books (pitch accent is shown in caps).